{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MCP LLMS-TXT Documentation Server\n",
    "\n",
    "## Overview\n",
    "\n",
    "[llms.txt](https://llmstxt.org/) is a website index for LLMs, providing background information, guidance, and links to detailed markdown files. IDEs like Cursor and Windsurf or apps like Claude Code/Desktop can use `llms.txt` to retrieve context for tasks. However, these apps use different built-in tools to read and process files like `llms.txt`. The retrieval process can be opaque, and there is not always a way to audit the tool calls or the context returned.\n",
    "\n",
    "[MCP](https://github.com/modelcontextprotocol) offers a way for developers to have *full control* over tools used by these applications. Here, we create [an open source MCP server](https://github.com/langchain-ai/mcpdoc) called `mcpdoc` to provide MCP host applications (e.g., Cursor, Windsurf, Claude Code/Desktop) with (1) a user-defined list of `llms.txt` files and (2) a simple  `fetch_docs` tool read URLs within any of the provided `llms.txt` files. This allows the user to audit each tool call as well as the context returned. \n",
    "\n",
    "![](https://github.com/user-attachments/assets/736f8f55-833d-4200-b833-5fca01a09e1b)\n",
    "\n",
    "## llms-txt\n",
    "\n",
    "You can find llms.txt files for langgraph and langchain here:\n",
    "\n",
    "| Library          | llms.txt                                                                                                   |\n",
    "|------------------|------------------------------------------------------------------------------------------------------------|\n",
    "| LangGraph Python | [https://langchain-ai.github.io/langgraph/llms.txt](https://langchain-ai.github.io/langgraph/llms.txt)     |\n",
    "| LangGraph JS     | [https://langchain-ai.github.io/langgraphjs/llms.txt](https://langchain-ai.github.io/langgraphjs/llms.txt) |\n",
    "| LangChain Python | [https://python.langchain.com/llms.txt](https://python.langchain.com/llms.txt)                             |\n",
    "| LangChain JS     | [https://js.langchain.com/llms.txt](https://js.langchain.com/llms.txt)                                     |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quickstart\n",
    "\n",
    "### Dry run at local\n",
    "\n",
    "#### Install uv\n",
    "Please see [official uv docs](https://docs.astral.sh/uv/getting-started/installation/#installation-methods) for other ways to install `uv`.\n",
    "\n",
    "```bash\n",
    "curl -LsSf https://astral.sh/uv/install.sh | sh\n",
    "```\n",
    "\n",
    "#### Choose an `llms.txt` file to use. \n",
    "For example, [here's](https://langchain-ai.github.io/langgraph/llms.txt) the LangGraph `llms.txt` file.\n",
    "\n",
    "> **Note: Security and Domain Access Control**\n",
    "> \n",
    "> For security reasons, mcpdoc implements strict domain access controls:\n",
    "> \n",
    "> 1. **Remote llms.txt files**: When you specify a remote llms.txt URL (e.g., `https://langchain-ai.github.io/langgraph/llms.txt`), mcpdoc automatically adds only that specific domain (`langchain-ai.github.io`) to the allowed domains list. This means the tool can only fetch documentation from URLs on that domain.\n",
    "> \n",
    "> 2. **Local llms.txt files**: When using a local file, NO domains are automatically added to the allowed list. You MUST explicitly specify which domains to allow using the `--allowed-domains` parameter.\n",
    "> \n",
    "> 3. **Adding additional domains**: To allow fetching from domains beyond those automatically included:\n",
    ">    - Use `--allowed-domains domain1.com domain2.com` to add specific domains\n",
    ">    - Use `--allowed-domains '*'` to allow all domains (use with caution)\n",
    "> \n",
    "> This security measure prevents unauthorized access to domains not explicitly approved by the user, ensuring that documentation can only be retrieved from trusted sources.]\n",
    "\n",
    "#### Test the MCP server locally with your `llms.txt` file(s) of choice:\n",
    "\n",
    "```bash\n",
    "uvx --from mcpdoc mcpdoc \\\n",
    "     --urls \"LangGraph:https://langchain-ai.github.io/langgraph/llms.txt\" \"LangChain:https://python.langchain.com/llms.txt\" \\\n",
    "     --transport sse \\\n",
    "     --port 8082 \\\n",
    "     --host localhost\n",
    "```\n",
    "\n",
    "> This command starts the MCP documentation server with the following parameters:\n",
    "> - `--from mcpdoc mcpdoc`: Installs and runs the mcpdoc package\n",
    "> - `--urls`: Specifies the llms.txt files to use, with labels \"LangGraph\" and \"LangChain\"\n",
    "> - `--transport sse`: Uses Server-Sent Events for communication\n",
    "> - `--port 8082`: Runs the server on port 8082\n",
    "> - `--host localhost`: Makes the server available on localhost\n",
    ">\n",
    "> The MCP server acts as a bridge between LLM applications and documentation sources, giving you full visibility and control over what context is being retrieved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "❯ uvx --from mcpdoc mcpdoc \\\n",
    "     --urls \"LangGraph:https://langchain-ai.github.io/langgraph/llms.txt\" \"LangChain:https://python.langchain.com/llms.txt\" \\\n",
    "     --transport sse \\\n",
    "     --port 8082 \\\n",
    "     --host localhost\n",
    "\n",
    "    ███╗   ███╗ ██████╗██████╗ ██████╗  ██████╗  ██████╗\n",
    "    ████╗ ████║██╔════╝██╔══██╗██╔══██╗██╔═══██╗██╔════╝\n",
    "    ██╔████╔██║██║     ██████╔╝██║  ██║██║   ██║██║\n",
    "    ██║╚██╔╝██║██║     ██╔═══╝ ██║  ██║██║   ██║██║\n",
    "    ██║ ╚═╝ ██║╚██████╗██║     ██████╔╝╚██████╔╝╚██████╗\n",
    "    ╚═╝     ╚═╝ ╚═════╝╚═╝     ╚═════╝  ╚═════╝  ╚═════╝\n",
    "\n",
    "\n",
    "Launching MCPDOC server with 2 doc sources\n",
    "INFO:     Started server process [1919]\n",
    "INFO:     Waiting for application startup.\n",
    "INFO:     Application startup complete.\n",
    "INFO:     Uvicorn running on http://localhost:8082 (Press CTRL+C to quit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run [MCP inspector](https://modelcontextprotocol.io/docs/tools/inspector) and connect to the running server:\n",
    "\n",
    "```bash\n",
    "npx @modelcontextprotocol/inspector\n",
    "\n",
    "Starting MCP inspector...\n",
    "⚙️ Proxy server listening on port 6277\n",
    "🔍 MCP Inspector is up and running at http://127.0.0.1:6274 🚀\n",
    "```\n",
    "\n",
    "![](https://github.com/user-attachments/assets/14645d57-1b52-4a5e-abfe-8e7756772704)\n",
    "\n",
    "> * Here, you can test the `Tools` calls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to Cursor \n",
    "\n",
    "* Open `Cursor Settings` and `MCP` tab.\n",
    "* This will open the `~/.cursor/mcp.json` file.\n",
    "\n",
    "![](https://github.com/user-attachments/assets/3d1c8eb3-4d40-487f-8bad-3f9e660f770a)\n",
    "\n",
    "* Paste the following into the file (we use the `langgraph-docs-mcp` name and link to the LangGraph `llms.txt`).\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"mcpServers\": {\n",
    "    \"langgraph-docs-mcp\": {\n",
    "      \"command\": \"uvx\",\n",
    "      \"args\": [\n",
    "        \"--from\",\n",
    "        \"mcpdoc\",\n",
    "        \"mcpdoc\",\n",
    "        \"--urls\",\n",
    "        \"LangGraph:https://langchain-ai.github.io/langgraph/llms.txt LangChain:https://python.langchain.com/llms.txt\",\n",
    "        \"--transport\",\n",
    "        \"stdio\"\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "* Confirm that the server is running in your `Cursor Settings/MCP` tab.\n",
    "* Best practice is to then update Cursor Global (User) rules.\n",
    "* Open Cursor `Settings/Rules` and update `User Rules` with the following (or similar):\n",
    "\n",
    "```\n",
    "for ANY question about LangGraph, use the langgraph-docs-mcp server to help answer -- \n",
    "+ call list_doc_sources tool to get the available llms.txt file\n",
    "+ call fetch_docs tool to read it\n",
    "+ reflect on the urls in llms.txt \n",
    "+ reflect on the input question \n",
    "+ call fetch_docs on any urls relevant to the question\n",
    "+ use this to answer the question\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test in Cursor\n",
    "\n",
    "* `CMD+L` (on Mac) to open chat.\n",
    "* Ensure `agent` is selected. \n",
    "\n",
    "![](https://github.com/user-attachments/assets/0dd747d0-7ec0-43d2-b6ef-cdcf5a2a30bf)\n",
    "\n",
    "Then, try an example prompt, such as:\n",
    "```\n",
    "what are types of memory in LangGraph?\n",
    "```\n",
    "\n",
    "![](https://github.com/user-attachments/assets/180966b5-ab03-4b78-8b5d-bab43f5954ed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Command-line Interface\n",
    "\n",
    "The `mcpdoc` command provides a simple CLI for launching the documentation server. \n",
    "\n",
    "You can specify documentation sources in three ways, and these can be combined:\n",
    "\n",
    "1. Using a YAML config file:\n",
    "\n",
    "* This will load the LangGraph Python documentation from the `sample_config.yaml` file in this repo.\n",
    "\n",
    "```bash\n",
    "mcpdoc --yaml sample_config.yaml\n",
    "```\n",
    "\n",
    "2. Using a JSON config file:\n",
    "\n",
    "* This will load the LangGraph Python documentation from the `sample_config.json` file in this repo.\n",
    "\n",
    "```bash\n",
    "mcpdoc --json sample_config.json\n",
    "```\n",
    "\n",
    "3. Directly specifying llms.txt URLs with optional names:\n",
    "\n",
    "* URLs can be specified either as plain URLs or with optional names using the format `name:url`.\n",
    "* You can specify multiple URLs by using the `--urls` parameter multiple times.\n",
    "* This is how we loaded `llms.txt` for the MCP server above.\n",
    "\n",
    "```bash\n",
    "mcpdoc --urls LangGraph:https://langchain-ai.github.io/langgraph/llms.txt --urls LangChain:https://python.langchain.com/llms.txt\n",
    "```\n",
    "\n",
    "You can also combine these methods to merge documentation sources:\n",
    "\n",
    "```bash\n",
    "mcpdoc --yaml sample_config.yaml --json sample_config.json --urls LangGraph:https://langchain-ai.github.io/langgraph/llms.txt --urls LangChain:https://python.langchain.com/llms.txt\n",
    "```\n",
    "\n",
    "## Additional Options\n",
    "\n",
    "- `--follow-redirects`: Follow HTTP redirects (defaults to False)\n",
    "- `--timeout SECONDS`: HTTP request timeout in seconds (defaults to 10.0)\n",
    "\n",
    "Example with additional options:\n",
    "\n",
    "```bash\n",
    "mcpdoc --yaml sample_config.yaml --follow-redirects --timeout 15\n",
    "```\n",
    "\n",
    "This will load the LangGraph Python documentation with a 15-second timeout and follow any HTTP redirects if necessary.\n",
    "\n",
    "## Configuration Format\n",
    "\n",
    "Both YAML and JSON configuration files should contain a list of documentation sources. \n",
    "\n",
    "Each source must include an `llms_txt` URL and can optionally include a `name`:\n",
    "\n",
    "### YAML Configuration Example (sample_config.yaml)\n",
    "\n",
    "```yaml\n",
    "# Sample configuration for mcp-mcpdoc server\n",
    "# Each entry must have a llms_txt URL and optionally a name\n",
    "- name: LangGraph Python\n",
    "  llms_txt: https://langchain-ai.github.io/langgraph/llms.txt\n",
    "```\n",
    "\n",
    "### JSON Configuration Example (sample_config.json)\n",
    "\n",
    "```json\n",
    "[\n",
    "  {\n",
    "    \"name\": \"LangGraph Python\",\n",
    "    \"llms_txt\": \"https://langchain-ai.github.io/langgraph/llms.txt\"\n",
    "  }\n",
    "]\n",
    "```\n",
    "\n",
    "## Programmatic Usage\n",
    "\n",
    "```python\n",
    "from mcpdoc.main import create_server\n",
    "\n",
    "# Create a server with documentation sources\n",
    "server = create_server(\n",
    "    [\n",
    "        {\n",
    "            \"name\": \"LangGraph Python\",\n",
    "            \"llms_txt\": \"https://langchain-ai.github.io/langgraph/llms.txt\",\n",
    "        },\n",
    "        # You can add multiple documentation sources\n",
    "        # {\n",
    "        #     \"name\": \"Another Documentation\",\n",
    "        #     \"llms_txt\": \"https://example.com/llms.txt\",\n",
    "        # },\n",
    "    ],\n",
    "    follow_redirects=True,\n",
    "    timeout=15.0,\n",
    ")\n",
    "\n",
    "# Run the server\n",
    "server.run(transport=\"stdio\")\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
